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DETAILED ACTION 

1 . This action is responsive to communications: Amendment filed 6/27/2006. 

2. Claims 1-28 are pending in the case. Claims 1, 16, and 22 are independent 
claims. 

Applicant's Response 

3. In Applicant's response dated 6/27/2006, Applicant amended independent claims 
1, 16, and 22. Applicant argued against all objections and rejection previously set forth 
in previous Office Action. 



r 
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Claim Rejections - 35 USC § 103 

4. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

5. Claims 1-23, 25, and 27 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Stubler et al. (US 6,804,684, B2; filed 5/7/2001) (hereinafter 
"Stubler") in view of Nolan et al. (Patent No.: 5,253,362; Date of Patent: Oct. 12, 
1993) (hereinafter "Nolan"). 

Regarding independent claim 1,16, and 22, Stubler teaches actively selecting 
examples of multimedia content to be annotated by a user and accepting input 
annotations from the user for the selected examples in fig. 2, fig. 6-7, col. 3 line 46 - col. 
4 line 12, col. 8 lines 18-23, and col. 9 line 65 - col. 10 line 18. Stubler teaches 
propagating the input annotations to other instances of multimedia content and storing 
the input annotations and the propagated annotations in fig. 2, col. 3 line 46 - col. 4 line 
12, and col. 8 lines 18-55. 

Stubler does not teach actively selecting examples of multimedia content to be 
annotated by a user, wherein the examples of multimedia content are selected based 
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on at least one criterion for achieving a maximal disambiguation result such that only 
those examples which are most ambiguous are selected. 

However Nolan teach actively selecting examples of multimedia content to be 
annotated by a user, wherein the examples of multimedia content are selected based 
on at least one criterion for achieving a maximal disambiguation result such that only 
those examples which are most ambiguous are selected (col. 5, lines 50-56; col. 6, lines 
13-21; Nolan teaches a list predefined commonly used annotation for which the nurse 
can select from. Using the broadest interpretation of Nolan's teaching, the Examiner 
concludes that the list of annotations would inherently be comprised of annotation 
examples, which are the most ambiguous.). 

Stubler teaches in col. 2 line 59 - col. 3 line 10 that it is directed towards 
propagating annotations from stored images to similar images that have not yet been 
annotated. Therefore, at the time of the invention it would have been obvious to a 
person of ordinary skill in the art to combine Stubler with Nolan's teaching of providing a 
list of commonly used annotation examples for the benefit of achieving a maximal 
disambiguation result such that only those examples which are most ambiguous are 
selected. 

Regarding dependent claims 2 and 17, Stubler teaches wherein the step of 
actively selecting is performed using a selection technique from the group consisting of 
deterministic and probabilistic in col. 4 line 64 - col. 5 line 19. 

Regarding dependent claims 3 and 18, Stubler teaches wherein the step of 
actively selecting which is performed deterministically or probabilistically, is based on 
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explicit models and feature proximity/similarity measures, and returns one or more 
examples of multimedia content to be annotated in fig. 2 and col. 8 lines 18-55. 

Regarding dependent claims 4 and 19, Stubler teaches wherein the step of 
actively selecting, which is performed deterministically or probabilistically, is based on 
implicit models and feature proximity/similarity measures, and returns one or more 
examples of multimedia content to be annotated in fig. 2 and col. 8 lines 18-55. 

Regarding dependent claims 5 and 20, Stubler teaches wherein an 
optimization criterion for active selection includes one or more criteria selected form the 
group consisting of: information measures and confidence in fig. 2 and col. 8 lines 18- 
55. 

Regarding dependent claims 6 and 21 , Stubler teaches wherein the 
multimedia content comprises one or more types selected from the group consisting of: 
images, audio, video, graphics, text, multimedia, Web pages, time series data, 
surveillance data, sensor data, relational data, and XML data in col. 3 line 46 - col. 4 
line 12. 

Regarding dependent claim 7, Stubler teaches wherein the input annotations 
are created by a user with reference to a vocabulary in col. 3 line 46 - col. 4 line 12 and 
col. 8 lines 18-23. 

Regarding dependent claim 8, Stubler teaches wherein the vocabulary 
contains one. or more items selected from the group consisting of: terms, concepts, 
labels, and annotations in col. 3 line 46 - col. 4 line 12 and col. 8 lines 18-23. 
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Regarding dependent claim 9, Stubler teaches wherein the process of creating 
input annotations by the user involves multimodal interaction with the user using 
graphical, textual, and/or speech interface in fig. 2 and col. 3 line 46 - col. 4 line 12. 

Regarding dependent claim 10, Stubler teaches wherein the input annotations 
are created by means of steps selected from the group consisting of: creating new 
annotations, deleting existing annotations, rejects proposed annotations, and modifying 
annotations in fig. 6-7 and col. 9 line 65 - col. 10 line 18. 

Regarding dependent claim 1 1 , Stubler teaches wherein the vocabulary is 
adaptively or dynamically organized and/or limited by the system of the user in fig. 6-7 
and col. 9 line 65 - col. 10 line 18. 

Regarding dependent claim 12, Stubler teaches wherein the multimodal 
interaction involves speech recognition, gaze detection, finger pointing, expression 
detection, and/or effective computing methods for sensing a user's state in fig. 6-7 and 
col. 9 line 65 -col. 10 line 18. 

Regarding dependent claim 13, Stubler teaches wherein the determination of 
the propagation of annotations is made deterministically or probabilistically and on the 
use of models for each annotation or for joint annotations in col. 4 line 64 - col. 5 line 
19. 

Regarding dependent claim 14, Stubler teaches wherein the models are 
created or learned automatically or semi-automatically and/or are updated adaptively 
from interaction with the user in fig. 2, col. 3 line 46 - col. 4 line 12, and col. 8 lines 18- 
55. 



Application/Control Number: 10/056,546 Page 7 

Art Unit: 2176 

Regarding dependent claim 15, Stubler teaches wherein the models are based 
on nearest neighbor voting or variants, parametric or statistical models, expert systems, 
rule-based systems, or hybrid techniques in fig. 2, col. 3 line 46 - col. 4 line 12, and col. 
8 lines 18-55. 

Regarding dependent claims 23, 25, and 27, Stubler does not teach wherein 
that at least one criterion includes an ambiguity level of the selected examples. 

However Nolan teach wherein that at least one criterion includes an ambiguity 
level of the selected examples.(co\. 5, lines 50-56; col. 6, lines 13-21; Nolan teaches a 
list predefined commonly used annotation for which the nurse can select from. Using 
the broadest interpretation of Nolan's teaching, the Examiner concludes that the 
composition of the list of annotations examples would inherently be comprised of at 
least one criterion that includes an ambiguity level of the selected examples.), 

Stubler teaches in col. 2 line 59 - col. 3 line 10 that it is directed towards 
propagating annotations from stored images to similar images that have not yet been 
annotated. Therefore, at the time of the invention it would have been obvious to a 
person of ordinary skill in the art to combine Stubler with Nolan's teaching of providing a 
list of commonly used annotation examples for the benefit of achieving a maximal 
disambiguation result such that only those examples which are most ambiguous are 
selected. 
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6. Claims 24, 26, and 28 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Stubler et al. (US 6,804,684, B2; filed 5/7/2001) (hereinafter 
"Stubler") in view of Nolan et al: (Patent No.: 5,253,362; Date of Patent: Oct. 12, 
1993) (hereinafter "Nolan") further in view Toyama, (US 6,816,847 B1 filed 
9/23/1999). 

Regarding dependent claims 24, 26, and 28, Stubler inview of Nolan does not 
teach wherein the at least one criterion includes a confidence level of the selected 
examples, the confidence level being inversely proportional to a distance of a new 
feature of. the selected examples from a separating hyperplane in an induced higher 
dimensional feature space. Toyama discloses the use of support vector machines in 
col. 6 lines 10-50. Toyama discloses wherein the at least one criterion includes a 
confidence level of the selected examples, the confidence level being inversely 
proportional to a distance of a new feature of the selected examples from a separating 
hyperplane in an induced higher dimensional feature space in col. 5 lines 15 - col. 6 line 
50. 

Stubler teaches in col. 2 line 59 - col. 3 line 10 that it is directed towards 
propagating annotations from stored images to similar images that have not yet been 
annotated. Toyama teaches in col. 1 line 47 - col. 2 line 14 that it is directed towards 
propagating annotations from a known training set to similar images that have do not 
have any associated annotations. Thus, the field of endeavor is very similar for Stubler 
in view of Nolan and Toyama. It would have been obvious to one of ordinary skill in the 
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art at the time the invention was made to have combined the teachings of Stubler in 
view of Nolan and Toyama to have created the claimed invention. It would have been 
obvious and desirable to have used the active selection techniques disclosed by 
Toyama in col. 5 lines 15-46 to have improved the active selection of images to be 
annotated as taught by Stubler in view of Nolan so that a maximal disambiguation result 
could have been achieved as is the reason for a Toyama selection of a wide variety of 
images in col. 5 lines 30-32 and a wide variety of annotations for the images in col. 5 
lines 32-46. 



Response to Arguments 

7. Applicant's arguments with respect to claims 1-28 have been considered but are 
moot in view of the new ground(s) of rejection. New ground(s) of rejection are based on 
newly found prior art reference of Nolan. An explanation of the rejection is given. 
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Conclusion 

8. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1 . 1 36(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the mailing date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to James J. Debrow whose telephone number is 571-272- 
5768. The examiner can normally be reached on 8:00-5:00pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Heather Herndon can be reached on 571-272-4136. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 
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